RECURRENT NEURAL NETWORK

Deep Bayesian Natural Language Processing (P19-4)

Challenge:	Existing methods for selecting training data for recurrent neural network language models are limited by computational costs.
Approach:	They propose an unsupervised importance sampling approach to selecting training data for recurrent neural network (RNN) language models.
Outcome:	The proposed approach outperforms models trained on the Billion Word and Wikitext-103 benchmark corpora.

Copied to clipboard

Jen-Tzung Chien

Identifying Emergent Research Trends by Key Authors and Phrases (C18-1)

Challenge:	Introduction to deep Bayesian learning for natural language addresses the fundamentals of statistical models and neural networks.
Approach:	This tutorial addresses the advances in deep Bayesian learning for natural language . it focuses on advanced Bayessian models and deep models . authors present case studies and domain applications to tackle different issues .
Outcome:	This tutorial focuses on advanced Bayesian models and deep models for natural language . case studies and domain applications are presented to tackle different issues in deep Bayessian processing, learning and understanding.

Copied to clipboard

Shenhao Jiang, Animesh Prasad, Min-Yen Kan, Kazunari Sugiyama

Neural Finite-State Transducers: Beyond Rational Relations (N19-1)

Challenge:	Existing methods to identify emergent research trends have been employed to generate corpora of large corporata.
Approach:	They propose an embedded trend detection framework which integrates hypothesis that important phrases are written by important authors within a field and vice versa.
Outcome:	The proposed framework outperforms baselines based on text centrality or citations over two large datasets of scientific articles.

Copied to clipboard

Chu-Cheng Lin, Hao Zhu, Matthew R. Gormley, Jason Eisner

Challenge:	a finite state transducer defines joint and conditional probability distributions over strings . a weighted finite-state transducers can only model certain functions, known as the rational relations .
Approach:	They propose a family of string transduction models defining joint and conditional probability distributions over pairs of strings.
Outcome:	The proposed models are more powerful than previous finite-state models with neural features.

PyOpenDial: A Python-based Domain-Independent Toolkit for Developing Spoken Dialogue Systems with Probabilistic Rules (D19-3)

Copied to clipboard

Youngsoo Jang, Jongmin Lee, Jaeyoung Park, Kyeng-Hun Lee, Pierre Lison, Kee-Eung Kim

Natural Language Interfaces

Challenge:	a recent development of spoken dialogue systems has enabled deep learning to achieve state-of-the-art performance.
Approach:	They propose a Python-based domain-independent, open-source toolkit for spoken dialogue systems.
Outcome:	The proposed toolkit extends OpenDial's Java-based architecture and provides new functions for neural dialogue state tracking and action planning.

A Lexicon-Based Supervised Attention Model for Neural Sentiment Analysis (C18-1)

Copied to clipboard

Yicheng Zou, Tao Gui, Qi Zhang, Xuanjing Huang

Nested Named Entity Recognition Revisited (N18-1)

Challenge:	Existing attention models do not take full advantage of sentiment lexicons, which provide rich sentiment information and play a critical role in sentiment analysis.
Approach:	They propose a lexicon-based supervised attention model which allows a neural network to focus on the sentiment content, thus generating sentiment-informative representations.
Outcome:	The proposed model outperforms existing models on three large-scale sentiment classification datasets.

Copied to clipboard

Arzoo Katiyar, Claire Cardie

Refining Source Representations with Relation Networks for Neural Machine Translation (C18-1)

Challenge:	Existing methods for named entity recognition and mention detection ignore nested entities along with any semantic relations between them.
Approach:	They propose a recurrent neural network-based approach to handle named entity recognition and nested entity mention detection simultaneously.
Outcome:	The proposed model outperforms state-of-the-art methods on three standard datasets.

Copied to clipboard

Wen Zhang, Jiawei Hu, Yang Feng, Qun Liu

The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain (2020.acl-main)

Challenge:	Existing neural machine translation frameworks that forget distant information and disregard relationship between source and target words are not effective.
Approach:	They propose to use relation networks to learn better representations of the source . they propose to associate source words with each other to help retain their relationships .
Outcome:	Experiments show that the proposed approach outperforms the encoder-decoder framework on several datasets.

Copied to clipboard

Annemarie Friedrich, Heike Adel, Federico Tomazic, Johannes Hingerl, Renou Benteau, Anika Marusczyk, Lukas Lange

Modeling Recurrence for Transformer (N19-1)

Challenge:	Using BERT embeddings leads to large performance gains, but with increasing task complexity, adding a recurrent neural network seems beneficial.
Approach:	They propose an annotation scheme for marking information on publications related to solid oxide fuel cells . they propose to use a recurrent neural network to solve a variety of tasks .
Outcome:	The proposed scheme is based on a corpus of 45 open-access scholarly articles and a neural network for a variety of tasks.

Copied to clipboard

Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, Zhaopeng Tu

Learning to Organize a Bag of Words into Sentences with Neural Networks: An Empirical Study (2021.naacl-main)

Challenge:	Existing studies show that the lack of recurrence modeling hinders the development of a translation model.
Approach:	They propose to model recurrence for Transformer with an additional recurrent encoder.
Outcome:	The proposed model outperforms the deep model on EnglishGerman and ChineseEnglish translation tasks.

Copied to clipboard

Chongyang Tao, Shen Gao, Juntao Li, Yansong Feng, Dongyan Zhao, Rui Yan

Challenge:	Existing approaches to encode natural languages without orders are lacking.
Approach:	They conduct a comprehensive analysis of the ability of neural models to organize sentences from a bag of words under three typical scenarios.
Outcome:	The proposed models can reorder or reconstruct sentences from a bag of words under three typical scenarios.

Detecting Cybersecurity Events from Noisy Short Text (N19-1)

Copied to clipboard

Semih Yagcioglu, Mehmet Saygin Seyfioglu, Begum Citamak, Batuhan Bardak, Seren Guldamlasioglu, Azmi Yuksel, Emin Islam Tatli

Recurrent Positional Embedding for Neural Machine Translation (D19-1)

Challenge:	Using domain-specific word embeddings, we propose a method to detect cyber security events from noisy short text.
Approach:	They propose a method that leverages domain-specific word embeddings and task-specific features to detect cyber security events from tweets.
Outcome:	The proposed model outperforms both baselines and traditional models on a dataset of 2K tweets and manually annotates them.

Copied to clipboard

Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita

Training-free Neural Architecture Search for RNNs and Transformers (2023.acl-long)

Challenge:	Existing translation systems that use positional embeddings only encode static order dependencies based on discrete numerical information, which may hinder the improvement of translation capacity.
Approach:	They propose a recurrent positional embedding approach based on word vectors that are learned by a neural network and integrated into existing multi-head self-attention models.
Outcome:	The proposed approach improves translation performance over the state-of-the-art Transformer baseline in English-to-German and NIST Chinese-to English translation tasks.

Copied to clipboard

Aaron Serianni, Jugal Kalita

On the Embeddings of Variables in Recurrent Neural Networks for Source Code (2021.naacl-main)

Challenge:	Neural architecture search (NAS) has allowed for the automatic creation of new and effective neural network architectures.
Approach:	They develop a new NAS metric that predicts the trained performance of an RNN architecture and significantly outperforms existing NAS metrics.
Outcome:	The proposed metric outperforms existing training-free metrics on the NAS-Bench-NLP benchmark.

Copied to clipboard

Nadezhda Chirkova

Disconnected Recurrent Neural Networks for Text Categorization (P18-1)

Challenge:	Source code processing heavily relies on the methods widely used in natural language processing (NLP) but requires specifics that need to be taken into account to achieve higher quality.
Approach:	They propose a recurrent mechanism that adjusts the learned semantics of a variable when it obtains more information about the variable’s role in the program.
Outcome:	The proposed method significantly improves the performance of the recurrent neural network, in code completion and bug fixing tasks.

Copied to clipboard

Baoxin Wang

Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning (P18-1)

Challenge:	Recurrent neural network (RNN) can model the entire sequence and capture long-term dependencies, but it does not do well in extracting key patterns.
Approach:	They propose a novel model which incorporates position-invariance into RNN and restricts the hidden state at each time step to represent words near the current position.
Outcome:	The proposed model improves on several benchmark datasets and achieves the best performance on several datasets.

Copied to clipboard

Hongge Chen, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Cho-Jui Hsieh

Multimodality

Challenge:	Visual language grounding is widely studied in modern neural image captioning systems . a novel algorithm for crafting adversarial examples in image captions is proposed .
Approach:	They propose an algorithm to craft adversarial examples in machine vision and perception . their approach provides two evaluation approaches to check if they can mislead systems .
Outcome:	The proposed algorithm can craft visually-similar adversarial examples with randomly targeted captions or keywords, and the results are transferable to other image captioning systems.

Automated Discovery of Mathematical Definitions in Text (2020.lrec-1)

Copied to clipboard

Natalia Vanetik, Marina Litvak, Sergey Shevchuk, Lior Reznik

Numerical Reasoning

Challenge:	a recent study shows that definition extraction is inefficient for one-sentence definitions . definitions are used in many automatic text analysis tasks, including ontology matching and construction .
Approach:	They propose to use convolutional neural network and recurrent neural network to identify mathematical definitions from one sentence.
Outcome:	The proposed dataset shows that deep learning methods can identify definitions from mathematical texts.

VCWE: Visual Character-Enhanced Word Embeddings (N19-1)

Copied to clipboard

Chi Sun, Xipeng Qiu, Xuanjing Huang

Visual Data in NLP

Challenge:	Currently, word embeddings are playing a pivotal role in many natural language processing tasks.
Approach:	They propose a model to learn Chinese word embeddings via three-level composition . they use convolutional neural network to extract intra-character compositionality from character shape .
Outcome:	The proposed model performs better on word similarity, sentiment analysis, named entity recognition and part-of-speech tagging tasks.

Semantic graph parsing with recurrent neural network DAG grammars (D19-1)

Copied to clipboard

Federico Fancellu, Sorcha Gilroy, Adam Lopez, Mirella Lapata

Multimodality

Challenge:	Semantic parsing is the task of mapping natural language to machine interpretable meaning representations.
Approach:	They propose a graph-aware sequence model that generates only well-formed graphs . their model is based on a multilingual semantic graphbank .
Outcome:	The proposed model yields competitive results in English and establishes the first results for German, Italian and Dutch.

Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models (2020.acl-main)

Copied to clipboard

Kaiji Lu, Piotr Mardziel, Klas Leino, Matt Fredrikson, Anupam Datta

Learning Universal Sentence Representations with Mean-Max Attention Autoencoder (D18-1)

Challenge:	LSTMs can capture syntactic rules in artificial languages, but it is unclear whether they are as capable in natural languages.
Approach:	They propose a causal account of structural properties as carried by paths across gates and neurons of a recurrent neural network that localizes and segments the concept into a set of gate or neuron-level paths.
Outcome:	The proposed model improves on a widely-studied multi-layer LSTM language model showing that it can learn subject-verb number agreement in English.

Copied to clipboard

Minghua Zhang, Yunfang Wu, Weikang Li, Wei Li

Direct Output Connection for a High-Rank Language Model (D18-1)

Challenge:	Existing methods to learn universal sentence representations focus on supervised learning.
Approach:	They propose a mean-max attention autoencoder that uses a multi-head mechanism to reconstruct the input sequence.
Outcome:	The proposed model outperforms state-of-the-art unsupervised single methods on a wide range of 10 transfer tasks.

Copied to clipboard

Sho Takase, Jun Suzuki, Masaaki Nagata

Recurrent Neural Language Models as Probabilistic Finite-state Automata (2023.emnlp-main)

Challenge:	Neural network language models have played a central role in recent natural language processing advances.
Approach:	They propose a recurrent neural network language model that combines probability distributions computed from a final RNN layer and also middle layers.
Outcome:	The proposed language model achieves the best score on the Penn Treebank and WikiText-2 benchmark datasets.

Copied to clipboard

Anej Svete, Ryan Cotterell

Neural Automated Essay Scoring Incorporating Handcrafted Features (2020.coling-main)

Challenge:	Existing studies have focused on the expressive power of recurrent neural network LMs to recognize unweighted formal languages.
Approach:	They propose to model a strict subset of probabilistic finite-state automata with RNNs . they show that an RNN requires left(N \|\|right) neurons to represent an LM .
Outcome:	The proposed language models can represent a strict subset of probabilistic distributions expressed by finite-state models.

Copied to clipboard

Masaki Uto, Yikuan Xie, Maomi Ueno

Dependency-based Mixture Language Models (2022.acl-long)

Challenge:	Automated essay scoring (AES) relies on handcrafted features, but recent studies have proposed a hybrid method that integrates handcrafted essay-level features into a DNN-AES model.
Approach:	They propose a method that integrates handcrafted features into a DNN-AES model.
Outcome:	The proposed method significantly improves the accuracy of existing methods.

Copied to clipboard

Zhixian Yang, Xiaojun Wan

Mama/Papa, Is this Text for Me? (2020.coling-main)

Challenge:	Existing models to incorporate syntactic structures into neural language models have relied heavily on elaborate components for a specific language model, which makes them unwieldy in practice to fit into other models.
Approach:	They propose a dependency-based mixture language model that incorporates syntactic structures into neural language models by mixing previous dependency modeling probabilities with self-attention.
Outcome:	The proposed method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks.

Copied to clipboard

Rashedur Rahman, Gwénolé Lecorvé, Aline Étienne, Delphine Battistelli, Nicolas Béchet, Jonathan Chevelu

Challenge:	Existing methods to predict minimal age from which text can be understood for children are unresolved in computational linguistics.
Approach:	They propose a method which predicts the minimum age from which a text can be understood by a recurrent neural network.
Outcome:	The proposed method outperforms state-of-the-art models at sentence and text levels and achieves mean absolute errors of 1.86 and 2.28.

Linguistic Versus Latent Relations for Modeling Coherent Flow in Paragraphs (D19-1)

Copied to clipboard

Dongyeop Kang, Eduard Hovy

Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation (P19-1)

Challenge:	a novel approach to paragraph planning involves a high-level control of different levels of relations between sentences . a proposed model with both forms of relations outperforms baselines in partially conditioned paragraph generation task .
Approach:	They propose two models that integrate human-created and latent relations into document-level language models . they focus on paragraph-level plan between sentences to produce coherent text .
Outcome:	The proposed models outperform baselines in partially conditioned paragraph generation task.

Copied to clipboard

Ning Dai, Jianze Liang, Xipeng Qiu, Xuanjing Huang

Regularized Context Gates on Transformer for Machine Translation (2020.acl-main)

Challenge:	Disentangling the content and style in the latent space is prevalent in text style transfer . recurrent neural networks (RNN) based encoder and decoder cannot deal with the long-term dependency .
Approach:	They propose a style transformer which disentangles style information in latent space . they propose encoding and decoding methods that disentangle style information .
Outcome:	The proposed method can achieve better style transfer and better content preservation.

Copied to clipboard

Xintong Li, Lemao Liu, Rui Wang, Guoping Huang, Max Meng